title	description	services	documentationcenter	tags	author	manager	editor	ms.assetid	ms.service	ms.devlang	ms.topic	ms.tgt_pltfrm	ms.workload	ms.date	ms.author
Create Windows-based Hadoop clusters in HDInsight using .NET SDK \| Microsoft Docs	Learn how to create HDInsight clusters for Azure HDInsight using .NET SDK.	hdinsight		azure-portal	mumian	jhubbard	cgronlun	134fbcdf-8f62-492f-84fd-8e2a3e0cd896	hdinsight	na	article	na	big-data	09/02/2016	jgao

Create Windows-based Hadoop clusters in HDInsight using .NET SDK

[!INCLUDE selector]

Learn how to create HDInsight clusters using .NET SDK. For other cluster creation tools and features click the tab select on the top of this page or see Cluster creation methods.

Prerequisites:

[!INCLUDE delete-cluster-warning]

Before you begin the instructions in this article, you must have the following:

An Azure subscription. See Get Azure free trial.
Visual Studio 2013 or 2015.

Access control requirements

[!INCLUDE access-control]

Create clusters

The HDInsight .NET SDK provides .NET client libraries that make it easier to work with HDInsight from a .NET Framework application. Follow the instructions below to create a Visual Studio console application and paste the code for creating a cluster.

The application requires an Azure resource group, and the default storage account. The Appendix A provides a PowerShell script to create the dependent components.

To create a Visual Studio console application

Create a new C# console application in Visual Studio.

Run the following Nuget command in the Nuget Package Management console.

 Install-Package Microsoft.Rest.ClientRuntime.Azure.Authentication -Pre
 Install-Package Microsoft.Azure.Management.ResourceManager -Pre
 Install-Package Microsoft.Azure.Management.HDInsight

From Solution Explorer, double-click Program.cs to open it, paste the following code, and provide values for the variables:

 using System;
 using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Rest;
 using Microsoft.Rest.Azure.Authentication;
 using Microsoft.Azure;
 using Microsoft.Azure.Management.HDInsight;
 using Microsoft.Azure.Management.HDInsight.Models;
 using Microsoft.Azure.Management.ResourceManager;
 using Microsoft.IdentityModel.Clients.ActiveDirectory;
 using System.Net.Http;

 namespace CreateHDInsightCluster
 {
     class Program
     {
         // The client for managing HDInsight
         private static HDInsightManagementClient _hdiManagementClient;
         // Replace with your AAD tenant ID if necessary
         private const string TenantId = UserTokenProvider.CommonTenantId; 
         private const string SubscriptionId = "<Your Azure Subscription ID>";
         // This is the GUID for the PowerShell client. Used for interactive logins in this example.
         private const string ClientId = "1950a258-227b-4e31-a9cf-717495945fc2";
         private const string ExistingResourceGroupName = "<Azure Resource Group Name>";
         private const string ExistingStorageName = "<Default Storage Account Name>.blob.core.windows.net";
         private const string ExistingStorageKey = "<Default Storage Account Key>";
         private const string ExistingBlobContainer = "<Default Blob Container Name>";
         private const string NewClusterName = "<HDInsight Cluster Name>";
         private const int NewClusterNumWorkerNodes = 2;
         private const string NewClusterLocation = "EAST US 2";     // Must be the same as the default Storage account
         private const OSType NewClusterOsType = OSType.Windows;
         private const string NewClusterType = "Hadoop";
         private const string NewClusterVersion = "3.2";
         private const string NewClusterUsername = "admin";
         private const string NewClusterPassword = "<HTTP User password>";

         static void Main(string[] args)
         {
             System.Console.WriteLine("Creating a cluster.  The process takes 10 to 20 minutes ...");

             // Authenticate and get a token
             var authToken = Authenticate(TenantId, ClientId, SubscriptionId);
             // Flag subscription for HDInsight, if it isn't already.
             EnableHDInsight(authToken);
             // Get an HDInsight management client
             _hdiManagementClient = new HDInsightManagementClient(authToken);

             // Set parameters for the new cluster
             var parameters = new ClusterCreateParameters
             {
                 ClusterSizeInNodes = NewClusterNumWorkerNodes,
                 UserName = NewClusterUsername,
                 Password = NewClusterPassword,
                 Location = NewClusterLocation,
                 DefaultStorageAccountName = ExistingStorageName,
                 DefaultStorageAccountKey = ExistingStorageKey,
                 DefaultStorageContainer = ExistingBlobContainer,
                 ClusterType = NewClusterType,
                 OSType = NewClusterOsType
             };
             // Create the cluster
             _hdiManagementClient.Clusters.Create(ExistingResourceGroupName, NewClusterName, parameters);

             System.Console.WriteLine("The cluster has been created. Press ENTER to continue ...");
             System.Console.ReadLine();
         }

         /// <summary>
         /// Authenticate to an Azure subscription and retrieve an authentication token
         /// </summary>
         /// <param name="TenantId">The AAD tenant ID</param>
         /// <param name="ClientId">The AAD client ID</param>
         /// <param name="SubscriptionId">The Azure subscription ID</param>
         /// <returns></returns>
         static TokenCloudCredentials Authenticate(string TenantId, string ClientId, string SubscriptionId)
         {
             var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + TenantId);
             var tokenAuthResult = authContext.AcquireToken("https://management.core.windows.net/", 
                 ClientId, 
                 new Uri("urn:ietf:wg:oauth:2.0:oob"), 
                 PromptBehavior.Always, 
                 UserIdentifier.AnyUser);
             return new TokenCloudCredentials(SubscriptionId, tokenAuthResult.AccessToken);
         }
         /// <summary>
         /// Marks your subscription as one that can use HDInsight, if it has not already been marked as such.
         /// </summary>
         /// <remarks>This is essentially a one-time action; if you have already done something with HDInsight
         /// on your subscription, then this isn't needed at all and will do nothing.</remarks>
         /// <param name="authToken">An authentication token for your Azure subscription</param>
         static void EnableHDInsight(TokenCloudCredentials authToken)
         {
             // Create a client for the Resource manager and set the subscription ID
             var resourceManagementClient = new ResourceManagementClient(new TokenCredentials(authToken.Token));
             resourceManagementClient.SubscriptionId = SubscriptionId;
             // Register the HDInsight provider
             var rpResult = resourceManagementClient.Providers.Register("Microsoft.HDInsight");
         }
     }
 }

Press F5 to run the application. A console window should open and display the status of the application. You will also be prompted to enter your Azure account credentials. It can take several minutes to create an HDInsight cluster.

Next steps

In this article, you have learned several ways to create an HDInsight cluster. To learn more, see the following articles:

Get started with Azure HDInsight - Learn how to start working with your HDInsight cluster
Run Hive jobs in HDInsight using .NET SDK
Run Pig jobs in HDInsight using .NET SDK
Run Sqoop jobs in HDInsight using .NET SDK
Run Oozie jobs in HDInsight
Azure HDInsight SDK documentation - Discover the HDInsight SDK

Appx-A Create dependent components

The following Azure PowerShell script can be use to create the dependent components needed by the .NET application in this tutorial.

[!INCLUDE upgrade-powershell]

####################################
# Set these variables
####################################
#region - used for creating Azure service names
$nameToken = "<Enter an Alias>" 
#endregion

####################################
# Service names and varialbes
####################################
#region - service names
$namePrefix = $nameToken.ToLower() + (Get-Date -Format "MMdd")

$resourceGroupName = $namePrefix + "rg"
$hdinsightClusterName = $namePrefix + "hdi"
$defaultStorageAccountName = $namePrefix + "store"
$defaultBlobContainerName = $hdinsightClusterName

$location = "East US 2"
#endregion

# Treat all errors as terminating
$ErrorActionPreference = "Stop"

####################################
# Connect to Azure
####################################
#region - Connect to Azure subscription
Write-Host "`nConnecting to your Azure subscription ..." -ForegroundColor Green
try{Get-AzureRmContext}
catch{Login-AzureRmAccount}
#endregion

#region - Create an HDInsight cluster
####################################
# Create dependent components
####################################
Write-Host "Creating a resource group ..." -ForegroundColor Green
New-AzureRmResourceGroup `
    -Name  $resourceGroupName `
    -Location $location

Write-Host "Creating the default storage account and default blob container ..."  -ForegroundColor Green
New-AzureRmStorageAccount `
    -ResourceGroupName $resourceGroupName `
    -Name $defaultStorageAccountName `
    -Location $location `
    -Type Standard_GRS

$defaultStorageAccountKey = (Get-AzureRmStorageAccountKey `
                                -ResourceGroupName $resourceGroupName `
                                -Name $defaultStorageAccountName)[0].Value
$defaultStorageContext = New-AzureStorageContext `
                                -StorageAccountName $defaultStorageAccountName `
                                -StorageAccountKey $defaultStorageAccountKey
New-AzureStorageContainer `
    -Name $defaultBlobContainerName `
    -Context $defaultStorageContext #use the cluster name as the container name

Write-Host "Use the following names in your .NET application" -ForegroundColor Green
Write-host "Resource Group Name: $resourceGroupName"
Write-host "Default Storage Account Name: $defaultStorageAccountName"
Write-host "Default Storage Account Key: $defaultStorageAccountKey"
Write-host "Default Blob Container Name: $defaultBlobContainerName"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hdinsight-hadoop-create-windows-clusters-dotnet-sdk.md

hdinsight-hadoop-create-windows-clusters-dotnet-sdk.md

Create Windows-based Hadoop clusters in HDInsight using .NET SDK

Prerequisites:

Access control requirements

Create clusters

Next steps

Appx-A Create dependent components

Files

hdinsight-hadoop-create-windows-clusters-dotnet-sdk.md

Latest commit

History

hdinsight-hadoop-create-windows-clusters-dotnet-sdk.md

File metadata and controls

Create Windows-based Hadoop clusters in HDInsight using .NET SDK

Prerequisites:

Access control requirements

Create clusters

Next steps

Appx-A Create dependent components