Skip to content

Commit

Permalink
Update outdated docs (WASB_ to BLOB_) (feathr-ai#850)
Browse files Browse the repository at this point in the history 
Signed-off-by: Jun Ki Min <[email protected]>

Signed-off-by: Jun Ki Min <[email protected]>
  • Loading branch information
@loomlike
loomlike authored Nov 11, 2022
1 parent b19480d commit 88a0348
Show file tree
Hide file tree
Showing 6 changed files with 23 additions and 23 deletions.
12 changes: 6 additions & 6 deletions docs/how-to-guides/feathr-configuration-and-env.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,14 @@ Feathr will get the configurations in the following order:
| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| SECRETS__AZURE_KEY_VAULT__NAME | Name of the Azure Key Vault service so that Feathr can get credentials from that service. | Optional |
| AZURE_CLIENT_ID | Client ID for authentication into Azure Services. Read [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) for more details. | This is required if you are using Service Principal to login with Feathr. |
| AZURE_TENANT_ID | Client ID for authentication into Azure Services. Read [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) for more details. | This is required if you are using Service Principal to login with Feathr. |
| AZURE_CLIENT_SECRET | Client ID for authentication into Azure Services. Read [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) for more details. | This is required if you are using Service Principal to login with Feathr. |
| AZURE_TENANT_ID | Tenant ID for authentication into Azure Services. Read [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) for more details. | This is required if you are using Service Principal to login with Feathr. |
| AZURE_CLIENT_SECRET | Client secret for authentication into Azure Services. Read [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) for more details. | This is required if you are using Service Principal to login with Feathr. |
| OFFLINE_STORE__ADLS__ADLS_ENABLED | Whether to enable ADLS as offline store or not. Available value: "True" or "False". Equivalent to "False" if not set. | Optional |
| ADLS_ACCOUNT | ADLS account that you connect to. | Required if using ADLS as an offline store. |
| ADLS_KEY | ADLS key that you connect to. | Required if using ADLS as an offline store. |
| OFFLINE_STORE__WASB__WASB_ENABLED | Whether to enable Azure BLOB storage as offline store or not. Available value: "True" or "False". Equivalent to "False" if not set. |
| WASB_ACCOUNT | Azure BLOB Storage account that you connect to. | Required if using Azure BLOB Storage as an offline store. |
| WASB_KEY | Azure BLOB Storage key that you connect to. | Required if using Azure BLOB Storage as an offline store. |
| BLOB_ACCOUNT | Azure BLOB Storage account that you connect to. | Required if using Azure BLOB Storage as an offline store. |
| BLOB_KEY | Azure BLOB Storage key that you connect to. | Required if using Azure BLOB Storage as an offline store. |
| S3_ACCESS_KEY | AWS S3 access key for the S3 account. | Required if using AWS S3 Storage as an offline store. |
| S3_SECRET_KEY | AWS S3 secret key for the S3 account. | Required if using AWS S3 Storage as an offline store. |
| OFFLINE_STORE__S3__S3_ENABLED | Whether to enable S3 as offline store or not. Available value: "True" or "False". Equivalent to "False" if not set. | Optional |
Expand Down Expand Up @@ -93,15 +93,15 @@ For example, if you want to use Feathr 0.9.0, you can set `os.environ["MAVEN_ART

## KAFKA_SASL_JAAS_CONFIG

Feathr uses Kafka behind the scene for streaming input, and Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL ([Simple Authentication and Security Layer](https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer)) configuration. You must provide JAAS configurations for all SASL authentication.
Feathr uses Kafka behind the scene for streaming input, and Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL ([Simple Authentication and Security Layer](https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer)) configuration. You must provide JAAS configurations for all SASL authentication.

For cloud services such as Azure EventHub or AWS Managed Streaming for Apache Kafka (MSK), they usually use `ConnectionString` as user name, and the password will be the exact content of the connection string. Feathr will automatically fill that part in so you don't have to worry about it.

In order to get the exact value of the `password` part (i.e. connection string), you can retrieve it from the Azure portal like below:

![EventHub Config](../images/eventhub_config.png)

For Azure EventHub, read [here](https://github.com/Azure/azure-event-hubs-for-kafka#updating-your-kafka-client-configuration) for how to get this string from the existing string in Azure Portal. The value will be something like: `Endpoint=sb://feathrazureci.servicebus.windows.net/;SharedAccessKeyName=feathrcipolicy;SharedAccessKey=aaaaaaaa=;EntityPath=feathrcieventhub`, and note that you don't need the `EntityPath=feathrcieventhub` part, as this represents the Kafka topic, which you will specify in the code in other places.
For Azure EventHub, read [here](https://github.com/Azure/azure-event-hubs-for-kafka#updating-your-kafka-client-configuration) for how to get this string from the existing string in Azure Portal. The value will be something like: `Endpoint=sb://feathrazureci.servicebus.windows.net/;SharedAccessKeyName=feathrcipolicy;SharedAccessKey=aaaaaaaa=;EntityPath=feathrcieventhub`, and note that you don't need the `EntityPath=feathrcieventhub` part, as this represents the Kafka topic, which you will specify in the code in other places.

So finally the configuration in Python will be something like:

Expand Down
8 changes: 4 additions & 4 deletions docs/samples/customer360/Customer360.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,8 @@
" - 'REDIS_PASSWORD'\n",
" - 'ADLS_ACCOUNT'\n",
" - 'ADLS_KEY'\n",
" - 'WASB_ACCOUNT'\n",
" - 'WASB_KEY'\n",
" - 'BLOB_ACCOUNT'\n",
" - 'BLOB_KEY'\n",
" - 'DATABRICKS_WORKSPACE_TOKEN_VALUE '\n",
" \n",
"offline_store:\n",
Expand Down Expand Up @@ -327,8 +327,8 @@
"os.environ['REDIS_PASSWORD'] = ''\n",
"os.environ['ADLS_ACCOUNT'] = ''\n",
"os.environ['ADLS_KEY'] = ''\n",
"os.environ['WASB_ACCOUNT'] = \"\"\n",
"os.environ['WASB_KEY'] = ''\n",
"os.environ['BLOB_ACCOUNT'] = \"\"\n",
"os.environ['BLOB_KEY'] = ''\n",
"os.environ['DATABRICKS_WORKSPACE_TOKEN_VALUE'] = ''"
]
},
Expand Down
10 changes: 5 additions & 5 deletions feathr_project/feathrcli/data/feathr_user_workspace/feathr_config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ project_config:
# the environemnt variables are optional, however you will need them if you want to use some of the services:
- ADLS_ACCOUNT
- ADLS_KEY
- WASB_ACCOUNT
- WASB_KEY
- BLOB_ACCOUNT
- BLOB_KEY
- S3_ACCESS_KEY
- S3_SECRET_KEY
- JDBC_TABLE
Expand All @@ -41,7 +41,7 @@ offline_store:
adls_enabled: true

# paths starts with wasb:// or wasbs://
# WASB_ACCOUNT and WASB_KEY should be set in environment variable
# BLOB_ACCOUNT and BLOB_KEY should be set in environment variable
wasb:
wasb_enabled: true

Expand Down Expand Up @@ -118,8 +118,8 @@ feature_registry:
delimiter: "__"
# controls whether the type system will be initialized or not. Usually this is only required to be executed once.
type_system_initialization: false


secrets:
azure_key_vault:
name: feathrazuretest3-kv
6 changes: 3 additions & 3 deletions feathr_project/test/test_user_workspace/feathr_config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ project_config:
# the environemnt variables are optional, however you will need them if you want to use some of the services:
- ADLS_ACCOUNT
- ADLS_KEY
- WASB_ACCOUNT
- WASB_KEY
- BLOB_ACCOUNT
- BLOB_KEY
- S3_ACCESS_KEY
- S3_SECRET_KEY
- JDBC_TABLE
Expand All @@ -41,7 +41,7 @@ offline_store:
adls_enabled: true

# paths starts with wasb:// or wasbs://
# WASB_ACCOUNT and WASB_KEY should be set in environment variable
# BLOB_ACCOUNT and BLOB_KEY should be set in environment variable
wasb:
wasb_enabled: true

Expand Down
4 changes: 2 additions & 2 deletions feathr_project/test/test_user_workspace/feathr_config_local.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ project_config:
# the environemnt variables are optional, however you will need them if you want to use some of the services:
- ADLS_ACCOUNT
- ADLS_KEY
- WASB_ACCOUNT
- WASB_KEY
- BLOB_ACCOUNT
- BLOB_KEY
- S3_ACCESS_KEY
- S3_SECRET_KEY
- JDBC_TABLE
Expand Down
6 changes: 3 additions & 3 deletions feathr_project/test/test_user_workspace/feathr_config_maven.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ project_config:
# the environemnt variables are optional, however you will need them if you want to use some of the services:
- ADLS_ACCOUNT
- ADLS_KEY
- WASB_ACCOUNT
- WASB_KEY
- BLOB_ACCOUNT
- BLOB_KEY
- S3_ACCESS_KEY
- S3_SECRET_KEY
- JDBC_TABLE
Expand All @@ -41,7 +41,7 @@ offline_store:
adls_enabled: true

# paths starts with wasb:// or wasbs://
# WASB_ACCOUNT and WASB_KEY should be set in environment variable
# BLOB_ACCOUNT and BLOB_KEY should be set in environment variable
wasb:
wasb_enabled: true

Expand Down

0 comments on commit 88a0348

Please sign in to comment.