Minor fixes to deployment README.md based on latest tryout, moved not…

…ebook to samples folder, more documentation cleanup. (feathr-ai#518) * Adding EventHub creation to the deployment script * Upgrading API versions and fixing the if clause of KV property creation for EH * Removing the need to ask for Principal Id from user and creating it in the template. Also removing bacpac file url parameter, and copying it to user's storage account as part of the template. This simplifies the flow for user * Making template more organized and updated documentaton * Addressing Blair's feedback to change registry image and app variable names * Fix for redirect URI, the callback one stopped working * Removing duplicate deployment text and adding reference to main guide so changes could be done only at single place * Adding parameters to support RBAC and passing them to app settings * Separating CLI and ARM documentation, adding bacpac import for RBAC, more bug fixes and doc change * Adding managed identity to webapp, instructions for Purview data curator role, consolidating bacpac into one * Fixing broken document link * Addressing PR feedback to remove commented code * Fixing broken links * Moving notebook under samples folder, minor updates to prod_reco advance notebook to get it working, and an extra permisison is required in notebook to submit job on Synapse * Adding instructions for steps to verify deployment was successful and run notebook samples * Removing feathr install from forked repo * Updating image * Fixing links, to get pytest check-link work. They would have worked but the absolute URL won't resolve until PR is merged. * Adding UI URL format
hubayirp · Jul 28, 2022 · cb4c847 · cb4c847
1 parent fcc8d35
commit cb4c847
Show file tree

Hide file tree

Showing 10 changed files with 24 additions and 898 deletions.
diff --git a/docs/README.md b/docs/README.md
@@ -129,7 +129,7 @@ Read [Point-in-time Correctness and Point-in-time Join in Feathr](https://linked
 
 ### Running Feathr Examples
 
-Follow the [quick start Jupyter Notebook](../feathr_project/feathrcli/data/feathr_user_workspace/product_recommendation_demo.ipynb) to try it out. There is also a companion [quick start guide](https://linkedin.github.io/feathr/quickstart_synapse.html) containing a bit more explanation on the notebook.
+Follow the [quick start Jupyter Notebook](./samples/product_recommendation_demo.ipynb) to try it out. There is also a companion [quick start guide](https://linkedin.github.io/feathr/quickstart_synapse.html) containing a bit more explanation on the notebook.
 
 ## 🗣️ Tech Talks on Feathr
 

diff --git a/docs/how-to-guides/azure-deployment-arm.md b/docs/how-to-guides/azure-deployment-arm.md
@@ -69,11 +69,20 @@ Feathr has native cloud integration and getting started with Feathr is very stra
 
     [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Flinkedin%2Ffeathr%2Fmain%2Fdocs%2Fhow-to-guides%2Fazure_resource_provision.json)
 
-3. If you are using Purview registry there is an additional step required for the deployment to work. Registry Server authenticates with Azure Purview using Managed Identity that was created by ARM template. The Managed Identity needs to be added to Azure Purview Collections as a __Data Curator__. For more details, please refer to [Access Control in Azure Documentation](https://docs.microsoft.com/en-us/azure/purview/catalog-permissions)
+3. If you are using Purview registry there is an additional step required for the deployment to work. Registry Server authenticates with Azure Purview using Managed Identity that was created by ARM template. The Managed Identity needs to be added to Azure Purview Collections as a __Data Curator__. For more details, please refer to [Access control in the Microsoft Purview governance portal](https://docs.microsoft.com/en-us/azure/purview/catalog-permissions)
     ![purview data curator role add](../images/purview_permission_setting.png)
 
     Only collection admins can perform the above operation, the user who created this Purview account is already one. If you want to add additional admins, you can do so by clicking on _Root collection permission_ option on Azure Purview page.
 
+Congratulations, you have successfully deployed Feathr on Azure. You can access your resources by going to the resource group that you created for the deployment. A good first test would be to access Feathr UI, you can access it by clicking on App Service URL. The URL would have the following format: https://{prefix}webapp.azurewebsites.net
+
+![app service url](../images/app-service-url.png)
+
+
+![feathr ui landing page](../images/feathr-ui-landingpage.png)
+
+## Next Steps
+Follow the quick start guide [here](https://linkedin.github.io/feathr/quickstart_synapse.html) to try out a notebook example.
 
 
 ## Known Issues/Workaround

diff --git a/docs/images/app-service-url.png b/docs/images/app-service-url.png
diff --git a/docs/images/feathr-ui-landingpage.png b/docs/images/feathr-ui-landingpage.png
diff --git a/docs/quickstart_synapse.md b/docs/quickstart_synapse.md
@@ -43,8 +43,7 @@ pip install git+https://github.com/linkedin/feathr.git#subdirectory=feathr_proje
 ```
 
 ## Step 3: Run the sample notebook
-
-We've provided a self-contained [sample notebook](https://github.com/linkedin/feathr/blob/main/feathr_project/feathrcli/data/feathr_user_workspace/product_recommendation_demo.ipynb) to act as the main content of this getting started guide. This documentation should be used more like highlights and further explanations of that demo notebook.
+We've provided a self-contained [sample notebook](./samples/product_recommendation_demo.ipynb) to act as the main content of this getting started guide. This documentation should be used more like highlights and further explanations of that demo notebook.
 
 ## Step 4: Update Feathr config
 
@@ -92,7 +91,7 @@ os.environ['ONLINE_STORE__REDIS__HOST'] = 'feathrazure.redis.cache.windows.net'
 
 ## Step 5: Setup environment variables.
 
-In the self-contained [sample notebook](https://github.com/linkedin/feathr/blob/main/feathr_project/feathrcli/data/feathr_user_workspace/product_recommendation_demo.ipynb), you also have to setup a few environment variables like below in order to access those cloud resources. You should be able to get those values from the first step.
+In the self-contained [sample notebook](./samples/product_recommendation_demo.ipynb), you also have to setup a few environment variables like below in order to access those cloud resources. You should be able to get those values from the first step.
 
 These values can also be retrieved by using cloud key value store, such as [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/):
 
@@ -187,7 +186,7 @@ client.multi_get_online_features("nycTaxiDemoFeature", ["239", "265"], ['f_locat
 
 ## Next steps
 
-- Run the [demo notebook](https://github.com/linkedin/feathr/blob/main/feathr_project/feathrcli/data/feathr_user_workspace/product_recommendation_demo.ipynb) to understand the workflow of Feathr.
+- Run the [demo notebook](./samples/product_recommendation_demo.ipynb) to understand the workflow of Feathr.
 - Read the [Feathr Documentation Page](https://linkedin.github.io/feathr/) page to understand the Feathr abstractions.
 - Read guide to understand [how to setup Feathr on Azure using Azure Resource Manager template](https://linkedin.github.io/feathr/how-to-guides/azure-deployment-arm.html).
 - Read guide to understand [how to setup Feathr step by step on Azure using Azure CLI](https://linkedin.github.io/feathr/how-to-guides/azure-deployment-cli.html).

diff --git a/docs/samples/customer360/Customer360.ipynb b/docs/samples/customer360/Customer360.ipynb
diff --git a/...rkspace/product_recommendation_demo.ipynb → ...samples/product_recommendation_demo.ipynb b/...rkspace/product_recommendation_demo.ipynb → ...samples/product_recommendation_demo.ipynb
@@ -29,19 +29,9 @@
       "source": [
         "## Prerequisite: Use Quick Start Template to Provision Azure Resources\n",
         "\n",
-        "Feathr has native cloud integration. To use Feathr on Azure, you only need three steps:\n",
+        "First step is to provision required cloud resources if you want to use Feathr. Feathr provides a python based client to interact with cloud resources.\n",
         "\n",
-        "- Get the `Principal ID` of your account by running `az ad signed-in-user show --query id -o tsv` in the link below (Select \"Bash\" if asked), and write down that value (something like `b65ef2e0-42b8-44a7-9b55-abbccddeefff`). Think this ID as something representing you when accessing Azure, and it will be used to grant permissions in the next step in the UI.\n",
-        "\n",
-        "[Launch Cloud Shell](https://shell.azure.com/bash)\n",
-        "\n",
-        "- Click the button below to deploy a minimal set of Feathr resources for demo purpose. You will need to fill in the `Principal ID` and `Resource Prefix`. You will need \"Owner\" permission of the selected subscription.\n",
-        "\n",
-        "[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Flinkedin%2Ffeathr%2Fmain%2Fdocs%2Fhow-to-guides%2Fazure_resource_provision.json)\n",
-        "\n",
-        "- Run the cells below.\n",
-        "\n",
-        "And the architecture is as below. In the above template, we are using Synapse as Spark provider, use Azure Data Lake Gen2 as offline store, and use Redis as online store, Azure Purview (Apache Atlas compatible) as feature reigstry. \n",
+        "Please follow the steps [here](https://linkedin.github.io/feathr/how-to-guides/azure-deployment-arm.html) to provision required cloud resources. Due to the complexity of the possible cloud environment, it is almost impossible to create a script that works for all the use cases. Because of this, [azure_resource_provision.sh](https://github.com/linkedin/feathr/blob/main/docs/how-to-guides/azure_resource_provision.sh) is a full end to end command line to create all the required resources, and you can tailor the script as needed, while [the companion documentation](https://linkedin.github.io/feathr/how-to-guides/azure-deployment-cli.html) can be used as a complete guide for using that shell script. \n",
         "\n",
         "\n",
         "![Architecture](https://github.com/linkedin/feathr/blob/main/docs/images/architecture.png?raw=true)"
@@ -144,17 +134,19 @@
         "\n",
         "**Permission**\n",
         "\n",
-        "To proceed with the following steps, you may need additional permission: permission to access the keyvault and permission to access the Storage Blob as a Contributor. Skip this step if you have already given yourself the access. Otherwise, run the following lines of command in the Cloud Shell before running the cell below.\n",
+        "To proceed with the following steps, you may need additional permission: permission to access the keyvault, permission to access the Storage Blob as a Contributor and permission to submit jobs to Synapse cluster. Skip this step if you have already given yourself the access. Otherwise, run the following lines of command in the Cloud Shell before running the cell below.\n",
         "\n",
         "~~~ \n",
         "userId=<email_id_of_account_requesting_access>\n",
         "resource_prefix=<resource_prefix>\n",
+        "synapse_workspace_name=\"${resource_prefix}syws\"\n",
         "keyvault_name=\"${resource_prefix}kv\"\n",
         "objectId=$(az ad user show --id $userId --query id -o tsv)\n",
         "az keyvault update --name $keyvault_name --enable-rbac-authorization false\n",
         "az keyvault set-policy -n $keyvault_name --secret-permissions get list --object-id $objectId\n",
         "az role assignment create --assignee $userId --role \"Storage Blob Data Contributor\"\n",
-        "~~~ \n"
+        "az synapse role assignment create --workspace-name $synapse_workspace_name --role \"Synapse Contributor\" --assignee $userId\n",
+        "~~~\n"
       ]
     },
     {

diff --git a/docs/samples/product_recommendation_demo_advanced.ipynb b/docs/samples/product_recommendation_demo_advanced.ipynb
diff --git a/feathr_project/feathrcli/data/feathr_user_workspace/nyc_driver_demo.ipynb b/feathr_project/feathrcli/data/feathr_user_workspace/nyc_driver_demo.ipynb
@@ -29,20 +29,9 @@
       "metadata": {},
       "source": [
         "## Prerequisite: Use Quick Start Template to Provision Azure Resources\n",
+        "First step is to provision required cloud resources if you want to use Feathr. Feathr provides a python based client to interact with cloud resources.\n",
         "\n",
-        "Feathr has native cloud integration. To use Feathr on Azure, you only need three steps:\n",
-        "\n",
-        "- Get the `Principal ID` of your account by running `az ad signed-in-user show --query id -o tsv` in the link below (Select \"Bash\" if asked), and write down that value (something like `b65ef2e0-42b8-44a7-9b55-abbccddeefff`). Think this ID as something representing you when accessing Azure, and it will be used to grant permissions in the next step in the UI.\n",
-        "\n",
-        "[Launch Cloud Shell](https://shell.azure.com/bash)\n",
-        "\n",
-        "- Click the button below to deploy a minimal set of Feathr resources for demo purpose. You will need to fill in the `Principal ID` and `Resource Prefix`. You will need \"Owner\" permission of the selected subscription.\n",
-        "\n",
-        "[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Flinkedin%2Ffeathr%2Fmain%2Fdocs%2Fhow-to-guides%2Fazure_resource_provision.json)\n",
-        "\n",
-        "- Run the cells below.\n",
-        "\n",
-        "And the architecture is as below. In the above template, we are using Synapse as Spark provider, use Azure Data Lake Gen2 as offline store, and use Redis as online store, Azure Purview (Apache Atlas compatible) as feature reigstry. \n",
+        "Please follow the steps [here](https://linkedin.github.io/feathr/how-to-guides/azure-deployment-arm.html) to provision required cloud resources. Due to the complexity of the possible cloud environment, it is almost impossible to create a script that works for all the use cases. Because of this, [azure_resource_provision.sh](https://github.com/linkedin/feathr/blob/main/docs/how-to-guides/azure_resource_provision.sh) is a full end to end command line to create all the required resources, and you can tailor the script as needed, while [the companion documentation](https://linkedin.github.io/feathr/how-to-guides/azure-deployment-cli.html) can be used as a complete guide for using that shell script.\n",
         "\n",
         "\n",
         "![Architecture](https://github.com/linkedin/feathr/blob/main/docs/images/architecture.png?raw=true)"