HiveServer2Connection.cursor has always user=None when called from ImpalaExecutionContext

[marqueewinq]

Steps to reproduce:

1. Create impala + LDAP stack with docker-compose

version: "3.5"

services:
  impala:
    image: codingtony/impala
    command: /start-bash.sh && /bin/true
    stdin_open: true
    tty: true
    privileged: true
    ports:
      - "9000:9000" 
      - "50010:50010" 
      - "50020:50020" 
      - "50070:50070" 
      - "50075:50075" 
      - "21000:21000" 
      - "21050:21050" 
      - "25000:25000" 
      - "25010:25010" 
      - "25020:25020"
    volumes:
      - ./etc.default.impala:/etc/default/impala
    depends_on:
      - ldap
  ldap:
    image: osixia/openldap:1.5.0
    privileged: true
    ports:
      - "389:389" 
      - "636:636" 
  ldap_admin:
    # access on 6443
    # echo "Login DN: cn=admin,dc=example,dc=org"
    # echo "Password: admin"
    image: osixia/phpldapadmin:0.9.0
    ports:
      - 6443:443
    environment:
      PHPLDAPADMIN_LDAP_HOSTS: ldap 
    depends_on:
      - ldap

2. Create the following script:

import logging

logging.basicConfig(level=logging.DEBUG)
from impala.dbapi import connect
from sqlalchemy import create_engine, inspect

host = "localhost"
port = "21050"
username = "admin"
password = "admin"
database = "default"
use_ssl = False
# auth_mechanism = "NOSASL" # or "LDAP"
auth_mechanism = "LDAP"

engine = create_engine(
    "impala://",
    connect_args={},
    creator=lambda: connect(
        host=host,
        port=port,
        database=database,
        timeout=5,
        user=username,
        password=password,
        use_ssl=use_ssl,
        auth_mechanism=auth_mechanism,
    ),
)
connection = engine.connect().execution_options(user=username)
inspector = inspect(connection)
table_names = inspector.get_table_names()
print(table_names)

3. Execute the script with python3 connect.py 2> >(grep 'req=TOpenSessionReq')

Expected output:

DEBUG:impala.hiveserver2:OpenSession: req=TOpenSessionReq(client_protocol=5, username='admin', password=None, configuration={})
[]

Actual output (notice the user name):

DEBUG:impala.hiveserver2:OpenSession: req=TOpenSessionReq(client_protocol=5, username='marqueewinq', password=None, configuration={})
[]

[marqueewinq]

With little pdb-ing i found that the cursor method of HiveServer2Connection does not receive the user argument from ImpalaExecutionContext.

I don't see the way to pass the user name from the script to the cursor method of HiveServer2Connection.

I'm not sure what would be the correct solution here; maybe read the user configuration from execution_options (configuration arg in cursor method)

[marqueewinq]

Monkey patch helps:

# connect.py
import logging

logging.basicConfig(level=logging.DEBUG)
from impala.dbapi import connect
from sqlalchemy import create_engine, inspect

from impala.sqlalchemy import ImpalaExecutionContext


def my_create_cursor(self):
    self._is_server_side = False
    cursor_configuration = self.execution_options.get("cursor_configuration", {})
    username = self.execution_options.get("user", None)
    return self._dbapi_connection.cursor(
        user=username, configuration=cursor_configuration
    )


ImpalaExecutionContext.create_cursor = my_create_cursor

host = "localhost"
port = "21050"
username = "admin"
password = "admin"
database = "default"
use_ssl = False
# auth_mechanism = "NOSASL" # or "LDAP"
auth_mechanism = "LDAP"

engine = create_engine(
    "impala://",
    connect_args={},
    creator=lambda: connect(
        host=host,
        port=port,
        database=database,
        timeout=5,
        user=username,
        password=password,
        use_ssl=use_ssl,
        auth_mechanism=auth_mechanism,
    ),
)
connection = engine.connect().execution_options(user=username)
inspector = inspect(connection)
table_names = inspector.get_table_names()
print(table_names)

Output:

$ python3 connect.py 2> >(grep 'req=TOpenSessionReq')
DEBUG:impala.hiveserver2:OpenSession: req=TOpenSessionReq(client_protocol=5, username='admin', password=None, configuration={})
[]

[marqueewinq]

I would happily create a PR with tests, but i need an advice from maintainers

• how exactly provide the system user for hive2 queries
• which tests would be acceptable in this case?