Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes and adds configurability to test_spark_sql_s3_with_privileges.py regtest #1060

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
+99 −66
Open

Fixes and adds configurability to test_spark_sql_s3_with_privileges.py regtest #1060

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
10c8755
Test fix and ability to pass REG_TEST_TOKEN
sfc-gh-tbowen Feb 24, 2025
0b9940e
Ability to configure bucket prefix
sfc-gh-tbowen Feb 24, 2025
291801f
Add reg_test env var to docker compose
sfc-gh-tbowen Feb 24, 2025
2f41a84
PR Comments
sfc-gh-tbowen Feb 26, 2025
05011bb
Add path prefix
sfc-gh-tbowen Feb 26, 2025
b082e16
Nit Updates
sfc-gh-tbowen Feb 27, 2025
fc66616
More updates
sfc-gh-tbowen Feb 27, 2025
319eca0
PR Feedback
sfc-gh-tbowen Feb 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
PR Feedback
  • Loading branch information
@sfc-gh-tbowen
sfc-gh-tbowen committed Feb 28, 2025
commit 319eca0132c9f7b363fe2f556fe7bd8b68865abb
13 changes: 8 additions & 5 deletions regtests/t_pyspark/src/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,27 +43,30 @@ def polaris_port():
@pytest.fixture
def polaris_path_prefix():
"""
Used to provide a path prefix between the port number and the standard polaris endpoint paths
Used to provide a path prefix between the port number and the standard polaris endpoint paths.
No leading or trailing /
:return:
"""
return os.getenv('POLARIS_PATH_PREFIX', '')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is expected to begin with / but to never end with it, while polaris_url_scheme and polaris_host are the opposite?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - you're right - there's some inconsistencies.
Polaris_Host => no leading or trailing /
AWS_Bucket_Prefix = > No leading or trailing /
URL Schema => Updated to not have leading or trailing /
POLARIS_PATH_PREFIX => Now also updated to not have a leading or trailing / (the callsites that use it can determine whether to add those)


@pytest.fixture
def polaris_url_scheme():
"""
Used to provide a path prefix between the port number and the standard polaris endpoint paths
The URL Schema - either http or https - no : or trailing /
:return:
"""
return os.getenv('POLARIS_URL_SCHEME', 'http://')
return os.getenv('POLARIS_URL_SCHEME', 'http')

@pytest.fixture
def polaris_url(polaris_url_scheme, polaris_host, polaris_port, polaris_path_prefix):
return f"{polaris_url_scheme}{polaris_host}:{polaris_port}{polaris_path_prefix}/api/management/v1"
polaris_path_prefix = polaris_path_prefix if len(polaris_path_prefix) == 0 else '/' + polaris_path_prefix
return f"{polaris_url_scheme}://{polaris_host}:{polaris_port}{polaris_path_prefix}/api/management/v1"


@pytest.fixture
def polaris_catalog_url(polaris_url_scheme, polaris_host, polaris_port, polaris_path_prefix):
return f"{polaris_url_scheme}{polaris_host}:{polaris_port}{polaris_path_prefix}/api/catalog"
polaris_path_prefix = polaris_path_prefix if len(polaris_path_prefix) == 0 else '/' + polaris_path_prefix
return f"{polaris_url_scheme}://{polaris_host}:{polaris_port}{polaris_path_prefix}/api/catalog"

@pytest.fixture
def test_bucket():
Expand Down