Skip to content

Commit

Permalink
QoL Improvements
Browse files Browse the repository at this point in the history
- DataLake containers now automatically created by ARM template to speed up lab execution;
- Instructions to creatd data lake containers removed from labs;
- Lab2 Mapping Data Flow Derived Column Transformation Script added;
- Lab5 Event Hub input screenshot updated.
  • Loading branch information
fabragaMS committed Jun 4, 2020
1 parent 9cd6837 commit 54cfe4e
Show file tree
Hide file tree
Showing 10 changed files with 263 additions and 207 deletions.
56 changes: 56 additions & 0 deletions Deploy/ADPE2EStop.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
Workflow Stop-Start-AzureVM
{
Param
(
[Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()]
[String]
$AzureSubscriptionId,
[Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()]
[String]
$ResourceGroupName,
[Parameter(Mandatory=$true)][ValidateSet("Start","Stop")]
[String]
$Action
)

$credential = Get-AutomationPSCredential -Name 'AzureCredential'
Login-AzureRmAccount -Credential $credential
Select-AzureRmSubscription -SubscriptionId $AzureSubscriptionId

if($AzureVMList -ne "All")
{
$AzureVMs = $AzureVMList.Split(",")
[System.Collections.ArrayList]$AzureVMsToHandle = $AzureVMs
}
else
{
$AzureVMs = (Get-AzureRmVM).Name
[System.Collections.ArrayList]$AzureVMsToHandle = $AzureVMs

}

foreach($AzureVM in $AzureVMsToHandle)
{
if(!(Get-AzureRmVM | ? {$_.Name -eq $AzureVM}))
{
throw " AzureVM : [$AzureVM] - Does not exist! - Check your inputs "
}
}

if($Action -eq "Stop")
{
Write-Output "Stopping VMs";
foreach -parallel ($AzureVM in $AzureVMsToHandle)
{
Get-AzureRmVM | ? {$_.Name -eq $AzureVM} | Stop-AzureRmVM -Force
}
}
else
{
Write-Output "Starting VMs";
foreach -parallel ($AzureVM in $AzureVMsToHandle)
{
Get-AzureRmVM | ? {$_.Name -eq $AzureVM} | Start-AzureRmVM
}
}
}
Binary file added Deploy/NYCDataSets/deploytoazure.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
146 changes: 135 additions & 11 deletions Deploy/azuredeploy.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,22 @@
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"parameters": {
"ADPAdminUserName": {
"type": "string",
"defaultValue": "adpadmin",
"metadata": {
"description": "Admin user name to connect to all services during the workshop."
}
},
"ADPAdminPassword": {
"type": "string",
"defaultValue": "P@ssw0rd123!",
"metadata": {
"description": "Admin password used to connect to all services during the workshop."
}
}
},
"variables": {
"ADPAdminUserName":"adpadmin",
"ADPAdminPassword":"P@ssw0rd123!",
Expand Down Expand Up @@ -224,8 +239,8 @@
},
"osProfile": {
"computerName": "[variables('ADPDesktopName')]",
"adminUsername": "[variables('ADPAdminUserName')]",
"adminPassword": "[variables('ADPAdminPassword')]",
"adminUsername": "[parameters('ADPAdminUserName')]",
"adminPassword": "[parameters('ADPAdminPassword')]",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": true
Expand Down Expand Up @@ -299,7 +314,7 @@
"tags": {},
"scale": null,
"properties": {
"isHnsEnabled": false,
"isHnsEnabled": true,
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [],
Expand All @@ -322,6 +337,115 @@
},
"dependsOn": []
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"sku": {
"name": "Standard_RAGRS",
"tier": "Standard"
},
"properties": {
"cors": {
"corsRules": []
},
"deleteRetentionPolicy": {
"enabled": false
}
}
},
{
"type": "Microsoft.Storage/storageAccounts/fileServices",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"sku": {
"name": "Standard_RAGRS",
"tier": "Standard"
},
"properties": {
"cors": {
"corsRules": []
}
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default/nycimagemetadata')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('ADPDataLakeStorageName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"properties": {
"publicAccess": "None"
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default/nycimages')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('ADPDataLakeStorageName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"properties": {
"publicAccess": "Blob"
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default/nyctaxidata-curated')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('ADPDataLakeStorageName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"properties": {
"publicAccess": "None"
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default/nyctaxidata-raw')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('ADPDataLakeStorageName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"properties": {
"publicAccess": "None"
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default/nysestocktrade')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('ADPDataLakeStorageName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"properties": {
"publicAccess": "None"
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(variables('ADPDataLakeStorageName'), '/default/polybase')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('ADPDataLakeStorageName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('ADPDataLakeStorageName'))]"
],
"properties": {
"publicAccess": "None"
}
},
{
"type": "Microsoft.Sql/servers",
"kind": "v12.0",
Expand All @@ -330,8 +454,8 @@
"location": "[resourceGroup().location]",
"scale": null,
"properties": {
"administratorLogin": "[variables('ADPAdminUserName')]",
"administratorLoginPassword": "[variables('ADPAdminPassword')]",
"administratorLogin": "[parameters('ADPAdminUserName')]",
"administratorLoginPassword": "[parameters('ADPAdminPassword')]",
"version": "12.0"
},
"dependsOn": [],
Expand Down Expand Up @@ -359,8 +483,8 @@
"location": "[resourceGroup().location]",
"scale": null,
"properties": {
"administratorLogin": "[variables('ADPAdminUserName')]",
"administratorLoginPassword": "[variables('ADPAdminPassword')]",
"administratorLogin": "[parameters('ADPAdminUserName')]",
"administratorLoginPassword": "[parameters('ADPAdminPassword')]",
"version": "12.0"
},
"dependsOn": [],
Expand Down Expand Up @@ -448,8 +572,8 @@
"storageKeyType": "SharedAccessKey",
"storageKey": "?sv=2018-03-28&ss=b&srt=sco&sp=rwl&se=2050-12-30T17:25:52Z&st=2019-04-05T09:25:52Z&spr=https&sig=4qrD8NmhaSmRFu2gKja67ayohfIDEQH3LdVMa2Utykc%3D",
"storageUri": "https://mdwresources.blob.core.windows.net/nycdatasets/NYCDataSets.bacpac",
"administratorLogin": "[variables('ADPAdminUserName')]",
"administratorLoginPassword": "[variables('ADPAdminPassword')]",
"administratorLogin": "[parameters('ADPAdminUserName')]",
"administratorLoginPassword": "[parameters('ADPAdminPassword')]",
"operationMode": "Import"
}
}
Expand Down
30 changes: 4 additions & 26 deletions Lab/Lab1/Lab1.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,21 +40,22 @@ In this section you are going to establish a Remote Desktop Connection to ADPDes
-------------|
**Execute these steps on your host computer**|

**IMPORTANT**: If you are executing the lab in a Spektra Labs environment, use the link provided in the Lab Environment tab to connect to the virtual machine. Once connected you can skip to the next section **Install required software onto ADPDesktop**.

1. In the Azure Portal, navigate to the lab resource group and click the **ADPDesktop** virtual machine.

2. On the ADPDesktop blade, from the Overview menu, click the Connect button.
2. On the ADPDesktop blade, from the Overview menu, click the Connect button.

![](./Media/Lab1-Image02.png)

3. On the **Connect to virtual machine** blade, click **Download RDP File**. This will download a .rdp file that you can use to establish a Remote Desktop Connection with the virtual machine.

![](./Media/Lab1-Image03.png)

4. If you have issues connecting via Remote Desktop Protocol (RDP), you can then connect via Azure Bastion by clicking the **Bastion** tab and providing the credentials indicated in the next section. This will open a new browser tab with the remote connection via SSL and HTML5.
4. If connected successfully using RDP, skip this step and go to the next section. If you faced any connectivity issues connecting via Remote Desktop Protocol (RDP), you can try connect via Azure Bastion by clicking the **Bastion** tab and providing the credentials indicated in the next section. This will open a new browser tab with the remote connection via SSL and HTML5.

![](./Media/Lab1-Image54.png)


## Install required software onto ADPDesktop
In this section you are going to install Power BI Desktop and Azure Data Studio on ADPDesktop.

Expand Down Expand Up @@ -158,29 +159,6 @@ with (distribution = round_robin)
go
```

## Create Staging Container on Azure Data Lake Storage Gen2
In this section you create a staging container in your data lake account that will be used as a staging environment for Polybase before data can be copied to Azure Synapse Analytics.

![](./Media/Lab1-Image24.jpg)

**IMPORTANT**|
-------------|
**Execute these steps on your host computer**|

1. In the Azure Portal, go to the lab resource group and locate the Azure Storage account **synapsedatalake*suffix***.

2. On the **Overview** panel, click **Containers**.

![](./Media/Lab1-Image25.png)

3. On the **synapsedatalake*suffix* – Containers** blade, click **+ Container**. On the **New container** blade, enter the following details:
<br>- **Name**: polybase
<br>- **Public access level**: Private (no anynymous access)

4. Click **OK** to create the new container.

![](./Media/Lab1-Image27.png)

## Create Azure Data Factory Pipeline to Copy Relational Data
In this section you will build an Azure Data Factory pipeline to copy a table from NYCDataSets database to Azure Synapse Analytics data warehouse.

Expand Down
Loading

0 comments on commit 54cfe4e

Please sign in to comment.