Lines 26-55 (alpha sort)

Author: v-thepet

articles/hdinsight/hdinsight-apache-spark-jupyter-notebook-install-locally.md

@@ -40,9 +40,9 @@ You  must install Python before you can install Jupyter notebooks. Both Python a
 
 1. Download the [Anaconda installer](https://www.continuum.io/downloads) for your platform and run the setup. While running the setup wizard, make sure you select the option to add Anaconda to your PATH variable.
 2. Run the following command to install Jupyter.
-   
+
         conda install jupyter
-   
+
     For more information on installting Jupyter, see [Installing Jupyter using Anaconda](http://jupyter.readthedocs.io/en/latest/install.html).
 
 ## Install the kernels and Spark magic
@@ -56,19 +56,19 @@ For clusters v3.5, please install sparkmagic 0.8.4 by executing `pip install spa
 In this section you configure the Spark magic that you installed earlier to connect to an Apache Spark cluster that you must have already created in Azure HDInsight.
 
 1. The Jupyter configuration information is typically stored in the users home directory. To locate your home directory on any OS platform, type the following commands.
-   
+
     Start the Python shell. On a command window, type the following:
-   
+
         python
-   
+
     On the Python shell, enter the following command to find out the home directory.
-   
+
         import os
         print(os.path.expanduser('~'))
 
 2. Navigate to the home directory and create a folder called **.sparkmagic** if it does not already exist.
 3. Within the folder, create a file called **config.json** and add the following JSON snippet inside it.
-   
+
         {
           "kernel_python_credentials" : {
             "username": "{USERNAME}",
@@ -83,7 +83,7 @@ In this section you configure the Spark magic that you installed earlier to conn
         }
 
 4. Substitute **{USERNAME}**, **{CLUSTERDNSNAME}**, and **{BASE64ENCODEDPASSWORD}** with appropriate values. You can use a number of utilities in your favorite programming language or online to generate a base64 encoded password for your actualy password. A simple Python snippet to run from your command prompt would be:
-   
+
         python -c "import base64; print(base64.b64encode('{YOURPASSWORD}'))"
 
 5. Configure the right Heartbeat settings in `config.json`:
@@ -100,16 +100,17 @@ In this section you configure the Spark magic that you installed earlier to conn
             "livy_server_heartbeat_timeout_seconds": 60,
             "heartbeat_retry_seconds": 1
 
-    >[!TIP] Heartbeats are sent to ensure that sessions are not leaked. Note that when a computer goes to sleep or is shut down, the hearbeat will not be sent, resulting in the session being cleaned up. For clusters v3.4, if you wish to disable this behavior, you can set the Livy config `livy.server.interactive.heartbeat.timeout` to `0` from the Ambari UI. For clusters v3.5, if you do not set the 3.5 configuration above, the session will not be deleted.
+    >[!TIP]
+    >Heartbeats are sent to ensure that sessions are not leaked. Note that when a computer goes to sleep or is shut down, the hearbeat will not be sent, resulting in the session being cleaned up. For clusters v3.4, if you wish to disable this behavior, you can set the Livy config `livy.server.interactive.heartbeat.timeout` to `0` from the Ambari UI. For clusters v3.5, if you do not set the 3.5 configuration above, the session will not be deleted.
 
 6. Start Jupyter. Use the following command from the command prompt.
-   
+
         jupyter notebook
 
 7. Verify that you can connect to the cluster using the Jupyter notebook and that you can use the Spark magic available with the kernels. Perform the following steps.
-   
+
    1. Create a new notebook. From the right hand corner, click **New**. You should see the default kernel **Python2** and the two new kernels that you install, **PySpark** and **Spark**.
-      
+
        ![Create a new Jupyter notebook](./media/hdinsight-apache-spark-jupyter-notebook-install-locally/jupyter-kernels.png "Create a new Jupyter notebook")
 
         Click **PySpark**.
@@ -122,7 +123,8 @@ In this section you configure the Spark magic that you installed earlier to conn
 
         If you can successfully retrieve the output, your connection to the HDInsight cluster is tested.
 
-    >[!TIP] If you want to update the notebook configuration to connect to a different cluster, update the config.json with the new set of values, as shown in Step 3 above. 
+    >[!TIP]
+    >If you want to update the notebook configuration to connect to a different cluster, update the config.json with the new set of values, as shown in Step 3 above.
 
 ## Why should I install Jupyter on my computer?
 There can be a number of reasons why you might want to install Jupyter on your computer and then connect it to a Spark cluster on HDInsight.
@@ -135,8 +137,8 @@ There can be a number of reasons why you might want to install Jupyter on your c
 
 > [!WARNING]
 > With Jupyter installed on your local computer, multiple users can run the same notebook on the same Spark cluster at the same time. In such a situation, multiple Livy sessions are created. If you run into an issue and want to debug that, it will be a complex task to track which Livy session belongs to which user.
-> 
-> 
+>
+>
 
 ## <a name="seealso"></a>See also
 * [Overview: Apache Spark on Azure HDInsight](hdinsight-apache-spark-overview.md)
@@ -162,4 +164,3 @@ There can be a number of reasons why you might want to install Jupyter on your c
 ### Manage resources
 * [Manage resources for the Apache Spark cluster in Azure HDInsight](hdinsight-apache-spark-resource-manager.md)
 * [Track and debug jobs running on an Apache Spark cluster in HDInsight](hdinsight-apache-spark-job-debugging.md)
-

articles/hdinsight/hdinsight-domain-joined-introduction.md

@@ -22,9 +22,9 @@ ms.author: saurinsh
 Azure HDInsight until today supported only a single user local admin. This worked great for smaller application teams or departments. As Hadoop based workloads gained more popularity in the enterprise sector, the need for enterprise grade capabilities like active directory based authentication, multi-user support, and role based access control became increasingly important. Using Domain-joined HDInsight clusters, you can create an HDInsight cluster joined to an Active Directory domain, configure a list of employees from the enterprise who can authenticate through Azure Active Directory to log on to HDInsight cluster. Anyone outside the enterprise cannot log on or access the HDInsight cluster. The enterprise admin can configure role based access control for Hive security using [Apache Ranger](http://hortonworks.com/apache/ranger/), thus restricting access to data to only as much as needed. Finally, the admin can audit the data access by employees, and any changes done to access control policies, thus achieving a high degree of governance of their corporate resources.
 
 > [!NOTE]
-> The new features described in this preview are available only on Linux-based HDInsight clusters for Hive workload. The other workloads, such as HBase, Spark, Storm and Kafka, will be enabled in future releases. 
-> 
-> 
+> The new features described in this preview are available only on Linux-based HDInsight clusters for Hive workload. The other workloads, such as HBase, Spark, Storm and Kafka, will be enabled in future releases.
+>
+>
 
 ## Benefits
 Enterprise Security contains four big pillars – Perimeter Security, Authentication, Authorization, and Encryption.
@@ -50,5 +50,4 @@ Protecting data is important for meeting organizational security and compliance
 * For configuring a Domain-joined HDInsight cluster, see [Configure Domain-joined HDInsight clusters](hdinsight-domain-joined-configure.md).
 * For managing a Domain-joined HDInsight clusters, see [Manage Domain-joined HDInsight clusters](hdinsight-domain-joined-manage.md).
 * For configuring Hive policies and run Hive queries, see [Configure Hive policies for Domain-joined HDInsight clusters](hdinsight-domain-joined-run-hive.md).
-* For running Hive queries using SSH on Domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](hdinsight-hadoop-linux-use-ssh-unix.md#connect-to-a-domain-joined-hdinsight-cluster).
-
+* For running Hive queries using SSH on Domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](hdinsight-hadoop-linux-use-ssh-unix.md#domain-joined).

articles/hdinsight/hdinsight-domain-joined-manage.md

@@ -31,14 +31,14 @@ A domain-joined HDInsight cluster has three new users in addition to Ambari Admi
 
 * **Ranger admin**:  This account is the local Apache Ranger admin account. It is not an active directory domain user. This account can be used to setup policies and make other users admins or delegated admins (so that those users can manage policies). By default, the username is *admin* and the password is the same as the Ambari admin password. The password can be updated from the Settings page in Ranger.
 * **Cluster admin domain user**: This account is an active directory domain user designated as the Hadoop cluster admin including Ambari and Ranger. You must provide this user’s credentials during cluster creation. This user has the following privileges:
-  
+
   * Join machines to the domain and place them within the OU that you specify during cluster creation.
-  * Create service principals within the OU that you specify during cluster creation. 
+  * Create service principals within the OU that you specify during cluster creation.
   * Create reverse DNS entries.
-    
-    Note the other AD users also have these privileges. 
-    
-    There are some end points within the cluster (for example, Templeton) which are not managed by Ranger, and hence are not secure. These end points are locked down for all users except the cluster admin domain user. 
+
+    Note the other AD users also have these privileges.
+
+    There are some end points within the cluster (for example, Templeton) which are not managed by Ranger, and hence are not secure. These end points are locked down for all users except the cluster admin domain user.
 * **Regular**: During cluster creation, you can provide multiple active directory groups. The users in these groups will be synced to Ranger and Ambari. These users are domain users and will have access to only Ranger-managed endpoints (for example, Hiveserver2). All the RBAC policies and auditing will be applicable to these users.
 
 ## Roles of Domain-joined HDInsight clusters
@@ -55,7 +55,7 @@ Domain-joined HDInsight have the following roles:
 1. Open the Ambari Management UI.  See [Open the Ambari Management UI](#open-the-ambari-management-ui).
 2. From the left menu, click **Roles**.
 3. Click the blue question mark to see the permissions:
-   
+
     ![Domain-joined HDInsight roles permissions](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-roles-permissions.png)
 
 ## Open the Ambari Management UI
@@ -64,36 +64,36 @@ Domain-joined HDInsight have the following roles:
 3. Click **Dashboard** from the top menu to open Ambari.
 4. Log on to Ambari using the cluster administrator domain user name and password.
 5. Click the **Admin** dropdown menu from the upper right corner, and then click **Manage Ambari**.
-   
+
     ![Domain-joined HDInsight manage Ambari](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-manage-ambari.png)
-   
+
     The UI looks like:
-   
+
     ![Domain-joined HDInsight Ambari management UI](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-ambari-management-ui.png)
 
 ## List the domain users synchronized from your Active Directory
 1. Open the Ambari Management UI.  See [Open the Ambari Management UI](#open-the-ambari-management-ui).
 2. From the left menu, click **Users**. You shall see all the users synced from your Active Directory to the HDInsight cluster.
-   
+
     ![Domain-joined HDInsight Ambari management UI list users](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-ambari-management-ui-users.png)
 
 ## List the domain groups synchronized from your Active Directory
 1. Open the Ambari Management UI.  See [Open the Ambari Management UI](#open-the-ambari-management-ui).
 2. From the left menu, click **Groups**. You shall see all the groups synced from your Active Directory to the HDInsight cluster.
-   
+
     ![Domain-joined HDInsight Ambari management UI list groups](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-ambari-management-ui-groups.png)
 
 ## Configure Hive Views permissions
 1. Open the Ambari Management UI.  See [Open the Ambari Management UI](#open-the-ambari-management-ui).
 2. From the left menu, click **Views**.
 3. Click **HIVE** to show the details.
-   
+
     ![Domain-joined HDInsight Ambari management UI Hive Views](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-ambari-management-ui-hive-views.png)
 4. Click the **Hive View** link to configure Hive Views.
 5. Scroll down to the **Permissions** section.
-   
+
     ![Domain-joined HDInsight Ambari management UI Hive Views configure permissions](./media/hdinsight-domain-joined-manage/hdinsight-domain-joined-ambari-management-ui-hive-views-permissions.png)
-6. Click **Add User** or **Add Group**, and then specify the users or groups that can use Hive Views. 
+6. Click **Add User** or **Add Group**, and then specify the users or groups that can use Hive Views.
 
 ## Configure users for the roles
  To see a list of roles and their permissions, see [Roles of Domain-joined HDInsight clusters](#roles-of-domain---joined-hdinsight-clusters).
@@ -105,5 +105,4 @@ Domain-joined HDInsight have the following roles:
 ## Next steps
 * For configuring a Domain-joined HDInsight cluster, see [Configure Domain-joined HDInsight clusters](hdinsight-domain-joined-configure.md).
 * For configuring Hive policies and run Hive queries, see [Configure Hive policies for Domain-joined HDInsight clusters](hdinsight-domain-joined-run-hive.md).
-* For running Hive queries using SSH on Domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](hdinsight-hadoop-linux-use-ssh-unix.md#connect-to-a-domain-joined-hdinsight-cluster).
-
+* For running Hive queries using SSH on Domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](hdinsight-hadoop-linux-use-ssh-unix.md#domain-joined).

articles/hdinsight/hdinsight-domain-joined-run-hive.md

@@ -28,16 +28,16 @@ Learn how to configure Apache Ranger policies for Hive. In this article, you cre
 ## Connect to Apache Ranger Admin UI
 **To connect to Ranger Admin UI**
 
-1. From a browser, connect to Ranger Admin UI. The URL is https://<ClusterName>.azurehdinsight.net/Ranger/. 
-   
+1. From a browser, connect to Ranger Admin UI. The URL is https://<ClusterName>.azurehdinsight.net/Ranger/.
+
    > [!NOTE]
    > Ranger uses different credentials than Hadoop cluster. To prevent browsers using cached Hadoop credentials, use new inprivate browser window to connect to the Ranger Admin UI.
-   > 
-   > 
+   >
+   >
 2. Log in using the cluster administrator domain user name and password:
-   
+
     ![HDInsight Domain-joined Ranger home page](./media/hdinsight-domain-joined-run-hive/hdinsight-domain-joined-ranger-home-page.png)
-   
+
     Currently, Ranger only works with Yarn and Hive.
 
 ## Create Domain users
@@ -51,23 +51,23 @@ In this section, you will create two Ranger policies for accessing hivesampletab
 1. Open Ranger Admin UI. See [Connect to Apache Ranger Admin UI](#connect-to-apache-ranager-admin-ui).
 2. Click **<ClusterName>_hive**, under **Hive**. You shall see two pre-configure policies.
 3. Click **Add New Policy**, and then enter the following values:
-   
+
    * Policy name: read-hivesampletable-all
    * Hive Database: default
    * table: hivesampletable
    * Hive column: *
    * Select User: hiveuser1
    * Permissions: select
-     
+
      ![HDInsight Domain-joined Ranger Hive policy configure](./media/hdinsight-domain-joined-run-hive/hdinsight-domain-joined-configure-ranger-policy.png).
-     
+
      > [!NOTE]
      > If a domain user is not populated in Select User, wait a few moments for Ranger to sync with AAD.
-     > 
-     > 
+     >
+     >
 4. Click **Add** to save the policy.
 5. Repeat the last two steps to create another policy with the following properties:
-   
+
    * Policy name: read-hivesampletable-devicemake
    * Hive Database: default
    * table: hivesampletable
@@ -98,20 +98,20 @@ In the last section, you have configured two policies.  hiveuser1 has the select
 
 1. Open a new or existing workbook in Excel.
 2. From the **Data** tab, click **From Other Data Sources**, and then click **From Data Connection Wizard** to launch the **Data Connection Wizard**.
-   
+
     ![Open data connection wizard][img-hdi-simbahiveodbc.excel.dataconnection]
 3. Select **ODBC DSN** as the data source, and then click **Next**.
 4. From ODBC data sources, select the data source name that you created in the previous step, and then  click **Next**.
 5. Re-enter the password for the cluster in the wizard, and then click **OK**. Wait for the **Select Database and Table** dialog to open. This can take a few seconds.
-6. Select **hivesampletable**, and then click **Next**. 
+6. Select **hivesampletable**, and then click **Next**.
 7. Click **Finish**.
-8. In the **Import Data** dialog, you can change or specify the query. To do so, click **Properties**. This can take a few seconds. 
+8. In the **Import Data** dialog, you can change or specify the query. To do so, click **Properties**. This can take a few seconds.
 9. Click the **Definition** tab. The command text is:
-   
+
        SELECT * FROM "HIVE"."default"."hivesampletable"
-   
+
    By the Ranger policies you defined,  hiveuser1 has select permission on all the columns.  So this query works with hiveuser1's credentials, but this query does not not work with hiveuser2's credentials.
-   
+
    ![Connection Properties][img-hdi-simbahiveodbc-excel-connectionproperties]
 10. Click **OK** to close the Connection Properties dialog.
 11. Click **OK** to close the **Import Data** dialog.  
@@ -121,23 +121,22 @@ To test the second policy (read-hivesampletable-devicemake) you created in the l
 
 1. Add a new sheet in Excel.
 2. Follow the last procedure to import the data.  The only change you will make is to use hiveuser2's credentials instead of hiveuser1's. This will fail because hiveuser2 only has permission to see two columns. You shall get the following error:
-   
+
         [Microsoft][HiveODBC] (35) Error from Hive: error code: '40000' error message: 'Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [hiveuser2] does not have [SELECT] privilege on [default/hivesampletable/clientid,country ...]'.
 3. Follow the same procedure to import data. This time, use hiveuser2's credentials, and also modify the select statement from:
-   
+
         SELECT * FROM "HIVE"."default"."hivesampletable"
-   
+
     to:
-   
+
         SELECT clientid, devicemake FROM "HIVE"."default"."hivesampletable"
-   
+
     When it is done, you shall see two columns of data imported.
 
 ## Next steps
 * For configuring a Domain-joined HDInsight cluster, see [Configure Domain-joined HDInsight clusters](hdinsight-domain-joined-configure.md).
 * For managing a Domain-joined HDInsight clusters, see [Manage Domain-joined HDInsight clusters](hdinsight-domain-joined-manage.md).
-* For running Hive queries using SSH on Domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](hdinsight-hadoop-linux-use-ssh-unix.md#connect-to-a-domain-joined-hdinsight-cluster).
+* For running Hive queries using SSH on Domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](hdinsight-hadoop-linux-use-ssh-unix.md#domain-joined).
 * For Connecting Hive using Hive JDBC, see [Connect to Hive on Azure HDInsight using the Hive JDBC driver](hdinsight-connect-hive-jdbc-driver.md)
 * For connecting Excel to Hadoop using Hive ODBC, see [Connect Excel to Hadoop with the Microsoft Hive ODBC drive](hdinsight-connect-excel-hive-odbc-driver.md)
 * For connecting Excel to Hadoop using Power Query, see [Connect Excel to Hadoop by using Power Query](hdinsight-connect-excel-power-query.md)
-