Skip to content

Latest commit

 

History

History
286 lines (246 loc) · 12.2 KB

hdinsight-hadoop-create-windows-clusters-arm-templates.md

File metadata and controls

286 lines (246 loc) · 12.2 KB
title description services documentationcenter tags author manager editor ms.assetid ms.service ms.devlang ms.topic ms.tgt_pltfrm ms.workload ms.date ms.author
Create Windows-based Hadoop clusters in HDInsight using Azure Resource Manager templates | Microsoft Docs
Learn how to create clusters for Azure HDInsight using Azure Resource Manager templates.
hdinsight
azure-portal
mumian
jhubbard
cgronlun
a4975787-699a-4cdc-b764-d8c9301e13ef
hdinsight
na
article
na
big-data
10/19/2016
jgao

Create Windows-based Hadoop clusters in HDInsight using Azure Resource Manager templates

[!INCLUDE selector]

Learn how to create HDInsight clusters using Azure Resource Manager templates. For more information, see Deploy an application with Azure Resource Manager template. For other cluster creation tools and features click the tab select on the top of this page or see Cluster creation methods.

Prerequisites:

[!INCLUDE delete-cluster-warning]

Before you begin the instructions in this article, you must have the following:

Access control requirements

[!INCLUDE access-control]

Resource Manager templates

Resource Manager template makes it easy to create HDInsight clusters, their dependent resources (such as the default storage account), and other resources (such as Azure SQL Database to use Apache Sqoop) for your application in a single, coordinated operation. In the template, you define the resources that are needed for the application and specify deployment parameters to input values for different environments. The template consists of JSON and expressions which you can use to construct values for your deployment.

A Resource Manager template for creating an HDInsight cluster and the dependent Azure Storage account can be found in Appendix-A. Use a text editor to save the template into a file on your workstation. You will learn how to call the template using various tools.

For more information about Resource Manager template, see

Deploy with PowerShell

The following procedure creates an HDInsight cluster.

To deploy a cluster using Resource Manager template

  1. Save the json file in Appendix A to your workstation.

  2. Set the parameters if needed.

  3. Run the template using the following PowerShell script:

     ####################################
 # Set these variables
 ####################################
 #region - used for creating Azure service names
 $nameToken = "<Enter an Alias>"
 $templateFile = "C:\HDITutorials-ARM\hdinsight-arm-template.json"
 #endregion

 ####################################
 # Service names and varialbes
 ####################################
 #region - service names
 $namePrefix = $nameToken.ToLower() + (Get-Date -Format "MMdd")

 $resourceGroupName = $namePrefix + "rg"
 $hdinsightClusterName = $namePrefix + "hdi"
 $defaultStorageAccountName = $namePrefix + "store"
 $defaultBlobContainerName = $hdinsightClusterName

 $location = "East US 2"

 $armDeploymentName = $namePrefix
 #endregion

 ####################################
 # Connect to Azure
 ####################################
 #region - Connect to Azure subscription
 Write-Host "`nConnecting to your Azure subscription ..." -ForegroundColor Green
 try{Get-AzureRmContext}
 catch{Login-AzureRmAccount}
 #endregion

 # Create a resource group
 New-AzureRmResourceGroup -Name $resourceGroupName -Location $Location

 # Create cluster and the dependent storage accounge
 $parameters = @{clusterName="$hdinsightClusterName";clusterStorageAccountName="$defaultStorageAccountName"}

 New-AzureRmResourceGroupDeployment `
     -Name $armDeploymentName `
     -ResourceGroupName $resourceGroupName `
     -TemplateFile $templateFile `
     -TemplateParameterObject $parameters

 # List cluster
 Get-AzureRmHDInsightCluster -ResourceGroupName $resourceGroupName -ClusterName $hdinsightClusterName

    The PowerShell script only configures the cluster name and the storage account name. You can set other values in the Resource Manager template.

For more information, see Deploy with PowerShell.

Deploy with Azure CLI

The following sample creates a cluster and its dependent storage account and container by calling a Resource Manager template:

azure login
azure config mode arm
azure group create -n hdi1229rg -l "East US 2"
azure group deployment create "hdi1229rg" "hdi1229" --template-file "C:\HDITutorials-ARM\hdinsight-arm-template.json" -p "{\"clusterName\":{\"value\":\"hdi1229win\"},\"clusterStorageAccountName\":{\"value\":\"hdi1229store\"},\"location\":{\"value\":\"East US 2\"},\"clusterLoginPassword\":{\"value\":\"Pass@word1\"}}"

Deploy with REST API

See Deploy with the REST API.

Deploy with Visual Studio

With Visual Studio, you can create a resource group project and deploy it to Azure through the user interface. You select the type of resources to include in your project and those resources are automatically added to Resource Manager template. The project also provides a PowerShell script to deploy the template.

For an introduction to using Visual Studio with resource groups, see Creating and deploying Azure resource groups through Visual Studio.

Next steps

In this article, you have learned several ways to create an HDInsight cluster. To learn more, see the following articles:

Appx-A: Resource Manager template

The following Azure Resource Manger template creates a Windows-based Hadoop cluster with the dependent Azure storage account.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
"parameters": {
    "location": {
    "type": "string",
    "defaultValue": "East US 2",
    "allowedValues": [
        "Central US",
        "North Europe",
        "East US",
        "East US 2",
        "North Central US",
        "South Central US",
        "West US",
        "North Europe",
        "West Europe",
        "East Asia",
        "Southeast Asia",
        "Japan East",
        "Japan West",
        "Brizil South",
        "Australia East",
        "Australia Southeast",
        "Central India"
    ],
    "metadata": {
        "description": "The location where all azure resources will be deployed."
    }
    },
    "clusterName": {
    "type": "string",
    "metadata": {
        "description": "The name of the HDInsight cluster to create."
    }
    },
    "clusterLoginUserName": {
    "type": "string",
    "defaultValue": "admin",
    "metadata": {
        "description": "These credentials can be used to submit jobs to the cluster and to log into cluster dashboards."
    }
    },
    "clusterLoginPassword": {
    "type": "securestring",
    "metadata": {
        "description": "The password for the cluster login."
    }
    },
    "clusterStorageAccountName": {
    "type": "string",
    "metadata": {
        "description": "The name of the storage account to be created and be used as the cluster's storage."
    }
    },
    "clusterStorageType": {
    "type": "string",
    "defaultValue": "Standard_LRS",
    "allowedValues": [
        "Standard_LRS",
        "Standard_GRS",
        "Standard_ZRS"
    ]
    },
    "clusterWorkerNodeCount": {
    "type": "int",
    "defaultValue": 4,
    "metadata": {
        "description": "The number of nodes in the HDInsight cluster."
    }
    }
},
    "variables": {},
    "resources": [
        {
        "name": "[parameters('clusterStorageAccountName')]",
        "type": "Microsoft.Storage/storageAccounts",
        "location": "[parameters('location')]",
        "apiVersion": "2015-05-01-preview",
        "dependsOn": [],
        "tags": {},
        "properties": {
            "accountType": "[parameters('clusterStorageType')]"
        }
        },
        {
        "name": "[parameters('clusterName')]",
        "type": "Microsoft.HDInsight/clusters",
        "location": "[parameters('location')]",
        "apiVersion": "2015-03-01-preview",
        "dependsOn": [
            "[concat('Microsoft.Storage/storageAccounts/',parameters('clusterStorageAccountName'))]"
        ],
        "tags": {},
        "properties": {
            "clusterVersion": "3.2",
            "osType": "Windows",
            "clusterDefinition": {
            "kind": "hadoop",
            "configurations": {
                "gateway": {
                "restAuthCredential.isEnabled": true,
                "restAuthCredential.username": "[parameters('clusterLoginUserName')]",
                "restAuthCredential.password": "[parameters('clusterLoginPassword')]"
                }
            }
            },
            "storageProfile": {
            "storageaccounts": [
                {
                "name": "[concat(parameters('clusterStorageAccountName'),'.blob.core.windows.net')]",
                "isDefault": true,
                "container": "[parameters('clusterName')]",
                "key": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('clusterStorageAccountName')), '2015-05-01-preview').key1]"
                }
            ]
            },
            "computeProfile": {
            "roles": [
                {
                "name": "headnode",
                "targetInstanceCount": "1",
                "hardwareProfile": {
                    "vmSize": "Large"
                }
                },
                {
                "name": "workernode",
                "targetInstanceCount": "[parameters('clusterWorkerNodeCount')]",
                "hardwareProfile": {
                    "vmSize": "Large"
                }
                }
            ]
            }
        }
        }
    ],
    "outputs": {
        "cluster": {
        "type": "object",
        "value": "[reference(resourceId('Microsoft.HDInsight/clusters',parameters('clusterName')))]"
        }
    }
}